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This study estimated the promotion power of public high schools in the District of Columbia. Promotion 
power is a measure of school effectiveness that distinguishes a school’s contributions to student 
outcomes from the contributions of the background characteristics of the students it serves. Promotion 


power scores are distinct from status measures such as graduation rate and college enrollment rate 
because they account for prior student achievement and other student background characteristics in 
measuring schools’ contributions. They complement value-added measures by using similar methods to 
examine additional, longer-term outcomes. The study found wide variation in high schools’ promotion 
power for college-ready SAT scores, high school graduation, and college enrollment. Schools with high 
promotion power for high school graduation were also more likely to have high promotion power for 
college enrollment. Student background characteristics were less strongly related to promotion power 
scores than to status measures, suggesting that high schools serving differing student populations can 
show strong promotion power. 


The Office of the State Superintendent of Education (OSSE) in the District of Columbia (DC), which oversees tra- 
ditional public and charter schools, expressed a need for high school accountability measures that fairly assess 
each school’s contribution to student outcomes. OSSE designs and administers an accountability system under 
its Every Student Succeeds Act (ESSA) plan. Currently, its high school accountability system does not include a 
growth metric for students, relying instead on status measures such as proficiency rate and graduation rate. 
These status measures, which can correlate with students’ family income, race/ethnicity, and other background 
characteristics outside a school’s control (Bailey & Dynarski, 2011), could penalize schools that serve students who 
are less prepared when they enter high school. Therefore, OSSE is interested in high school accountability mea- 
sures that identify a school’s contributions to student outcomes independent of the advantages or disadvantages 
of the students it serves. 


This study explored promotion power, a measure of school effectiveness that separates a school’s contributions 
to student outcomes from the contributions of the background characteristics of the students it serves (Deutsch 
et al., 2020). Specifically, this study examined public DC high schools’ power to promote college- and career- 
ready SAT scores, as defined by the College Board’s benchmarks (college-ready SAT scores); high school gradua- 
tion; and college enrollment. Promotion power scores are distinct from status measures such as graduation rate 
and college enrollment rate because they account for prior student achievement and other student background 
characteristics in measuring schools’ contributions. They complement value-added measures by using similar 
methods to examine additional, longer-term outcomes. 


Promotion power models are applicable beyond DC. All states use high school 
graduation as an accountability measure in their ESSA plans and could benefit 
from an approach that measures each school’s contribution to its students’ grad- 
uation rate. In addition to accountability, promotion power models could also 
help OSSE and other states and school districts identify schools that require addi- 
tional support and schools that could serve as models for improving students’ 
long-term success. 


For additional information, 
including technical 
methods and supporting 


analyses, access the 
report appendixes at 
https://go.usa.gov/xF VxQ. 
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This study also sought to understand whether schools with high promotion power for college-ready SAT scores 
also have high promotion power for other critical indicators of student success. Although many accountability 
systems assess school performance using students’ achievement on standardized tests, growing evidence con- 
firms that test scores do not capture all the ways in which a school influences student outcomes (Jackson et al., 
2020). Filling a gap in the literature, this study explored not only schools’ promotion power for college-ready SAT 
scores, high school graduation, and college enrollment outcomes individually, but also the relationship between 
the three (Jennings et al., 2015). This exercise also served as a check on the validity of promotion power models, 
as it could be concerning if promotion power scores were negatively correlated across outcomes. 


Research questions 
This study addressed three research questions about the promotion power of public DC high schools: 


1. Do DC high schools differ in their power to promote college-ready SAT scores, high school graduation, and 
college enrollment? 


2. What is the relationship between high schools’ promotion power for college-ready SAT scores and their 
promotion power for high school graduation and college enrollment? 


3. How strongly related are promotion power scores and student background characteristics, and how does this 
compare with the relationship between student background characteristics and status measures such as high 
school graduation rate and college enrollment rate? 


To answer these research questions, the study team used three types of variables: 


e OSSE administrative records, to construct variables representing the three outcomes of interest (table 1; see 
box 1 for asummary of the data and methods and appendix A for more detail). 


e Student background characteristics, to account for the different population of students that each high school 
serves. Different high schools serve students who might be more likely or less likely than other student popula- 
tions to achieve long-term outcomes. For example, the promotion power models accounted for students’ prior 
achievement on the grade 8 Partnership for Assessment of Readiness for College and Careers test (an annual 
standardized test of math and English language arts), demographic characteristics, and other characteristics 
measured before high school, such as receiving special education services in middle school. 


¢ Enrollment data from the school or schools that each student attended, to link students to specific high schools. 


All variables are described further in appendix A. 
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Table 1. Outcomes analyzed in the promotion power models for public high schools in the District of Columbia (DC) 


Percentage of students 


in the sample who 


Outcome Titel (eth xelmel-troglolateya) met the outcome 
College-ready SAT scores Whether students’ SAT math and reading scores met the College Board’s college 17 

and career readiness benchmarks: scores of 530 in math and 480 in reading? 
High school graduation Whether students graduated high school within four years of beginning high school 76 
College enrollment Whether students enrolled in a two-year or a four-year college in the fall semester 47 


four years after beginning grade 9 


Note: n = 3,568 first-time grade 9 students in public DC high schools during the 2015/16 school year. 


a. For students who took the SAT more than once, this indicator measures whether these students ever scored above 530 in math and 480 in reading, 
across all available SAT administrations. For example, students were considered to have a college-ready SAT score if they scored above 530 in math but 
below 480 in reading the first time they took the SAT and above 480 in reading the second time. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education administrative records for SAT scores and high school 
graduation rates and National Student Clearinghouse records for college enrollment data. 


Box 1. Data sources, sample, and methods 


Data sources. The data for this study came from three sources: administrative data from the DC Office of the State Superinten- 
dent of Education (OSSE), National Student Clearinghouse data shared by OSSE, and publicly available data from the U.S. Depart- 
ment of Education’s Civil Rights Data Collection (CRDC; http://ocrdata.ed.gov). Data on student background characteristics, high 
school outcomes, and school enrollment information linking students to schools came from OSSE administrative records. College 
enrollment information also came from OSSE but originated at the National Student Clearinghouse. Data on school-level suspen- 
sions from students’ middle schools came from the CRDC. All background characteristics from OSSE and the CRDC were measured 
before students entered grade 9. Details on the specific measures used in the analyses are in appendix A. 


Sample. The analytic sample for all three research questions included 3,568 students who were first-time grade 9 students at any 
of 36 public DC high schools (16 district operated and 20 charter) during the 2015/16 school year. The analytic sample excluded 
students who did not have grade 8 test score data—a critical background characteristic in the analysis—and students who entered 
a public DC high school after grade 9. The sample included students who dropped out or transferred out of public DC high schools. 
Detailed information on the analytic sample is in appendix A. 


Methodology. To calculate promotion power scores, the study team used a fixed effects regression model with a fixed effect for 
each high school. Heuristically, the model predicts the chances that each student would achieve an outcome (for example, high 
school graduation) based on the student’s prior achievement and other background characteristics. It then compares the average 
actual outcome among all students in a high school with the average outcomes predicted by the model. The promotion power 
score is the difference between the actual outcome and the predicted outcome.' 

Promotion power scores are constructed to have an average of O so that a score below O indicates that the school was less 
effective than the average school at promoting the outcome and a score above 0 indicates that the school was more effective than 
the average school. For example, a promotion power score of .10 on the college-ready SAT score outcome means that, on average, 
the school increased the likelihood that its students would meet the college and career readiness benchmarks on the SAT by 10 
percentage points relative to their expected achievement if they had attended the average school. Likewise, a promotion power 
score of —.10 indicates that the school’s students were 10 percentage points less likely to meet those benchmarks compared with 
their expected achievement if they had attended the average school. 


Research question 1. To assess the extent to which promotion power varied across high schools, the study team used the 
benchmarks for effect sizes of education interventions introduced in Kraft (2019): less than 0.05 is small, 0.05 to less than 0.20 is 
medium, and 0.20 or greater is large. Differences in promotion power scores across schools were classified as large if the standard 
deviation of promotion power scores, in student standard deviation units, was 0.20 or greater. 

To more clearly illustrate the distribution of school promotion power, the report shows how much students’ chances of 
achieving a certain outcome, such as graduating high school, would change if they attended a school with high promotion power 
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or low promotion power instead of an average school. The team used the 90th percentile to indicate schools with high promotion 
power and the 10th percentile to indicate schools with low promotion power. 

A final way of illustrating the distribution of promotion power and the precision of the estimates was to examine the propor- 
tion of schools with scores that were statistically higher or lower than the average, using a 95 percent confidence interval. 


Research question 2. To determine the relationship between schools’ power to promote college-ready SAT scores and their power to 
promote students’ high school graduation and college enrollment, the study team estimated the correlation between schools’ promo- 
tion power scores for college-ready SAT scores and their promotion power scores for high school graduation and college enrollment. 


Research question 3. To examine how promotion power scores compare to status measures, the study team assessed whether 
grade 8 test scores have a weaker relationship with promotion power scores for a given outcome than the status measure for that 
outcome. For each outcome the team compared two relationships: the correlation between the average grade 8 achievement of 
a high school’s students and the school’s promotion power score, and the correlation between average grade 8 achievement and 
the school’s status measure, or average outcome. The study used the average of grade 8 math and English language arts standard- 
ized test scores (z-scores) as an indicator for the achievement level of incoming students. 
A more detailed description of the methodology for each research question is available in appendix A. 
Note 


1. For heuristic purposes the estimation is described as occurring in multiple, distinct steps. In fact, however, the estimation occurred in a single step in 
the fixed-effect regression framework. 


Findings 
Public high schools in the District of Columbia varied in their power to promote student outcomes 


This section examines three indicators to assess the extent to which promotion power scores meaningfully distin- 
guish among high schools. 


The distribution of promotion power scores reveals wide variation in high schools’ power to promote student out- 
comes. Schools’ promotion power for college-ready SAT scores, high school graduation, and college enrollment 
varied, as displayed in figure 1. Each blue dot in the figure represents an individual high school’s promotion power 
score, and the black bars indicate the 95 percent confidence interval. For each outcome many schools were clus- 
tered close to the average of O, as expected, but some schools were more effective or less effective than the 
average school by at least 10 percentage points. 


The variation in promotion power across schools was large. The standard deviation of promotion power scores, in 
student standard deviation units, was 0.33 for college-ready SAT scores, 0.22 for high school graduation, and 0.25 
for college enrollment (table 2). For all three outcomes the effect size was large (a standard deviation of 0.20 or 
greater, according to the threshold for effect sizes of education interventions introduced in Kraft, 2019)." 


Compared with similar students attending schools with average promotion power, students attending high schools 
with high promotion power are expected to have better outcomes and students attending schools with low pro- 
motion power are expected to have worse outcomes. Relative to attending a high school with average promo- 
tion power for college-ready SAT scores, the model predicts that an average student attending a high school at 
the 90th percentile for college-ready SAT promotion power was 19 percentage points more likely to achieve a 
college-ready SAT score (a 36 percent chance of achieving the benchmarks at a school with high promotion power 


1. These standard deviations are larger than those reported in studies that measured teacher value-added and concluded that there 
were meaningful differences across teachers (for example, Kane & Staiger, 2008, and Chetty et al., 2014, both of which found standard 
deviations of less than 0.15). They are in the same range or larger than the standard deviations found in a study of the promotion 
power of Louisiana high schools (Deutsch et al., 2020), which ranged from 0.14 to 0.24 for similar outcomes. 
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Figure 1. Distribution of promotion power scores across public high schools in the District of Columbia (DC), by 
outcome 
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Note: n = 3,568 first-time grade 9 students in 36 public DC high schools during the 2015/16 school year. Each blue dot represents a high school’s promotion 
power score. The black bars indicate the 95 percent confidence interval. The horizontal orange line at O represents the promotion power of the average school. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


Table 2. Variation in promotion power scores across public high schools in the District of Columbia (DC), by 
outcome (student standard deviation units) 


Measure College-ready SAT score High school graduation College enrollment 


Standard deviation 0.33 0.22 0.25 


Note: n = 3,568 first-time grade 9 students in 36 public DC high schools during the 2015/16 school year. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


versus a17 percent chance at a school with average promotion power; figure 2). Similarly, the model predicts that 
an average student attending a high school with average promotion power for graduating high school would have 
a 76 percent chance of graduating within four years. For that same average student attending a school at the 90th 
percentile of promotion power for high school graduation, the model predicts an 86 percent chance of graduat- 
ing high school, a difference of 10 percentage points. And relative to attending a high school with average pro- 
motion power for college enrollment, the model predicts that an average student attending a high school at the 
90th percentile of college enrollment promotion power was 16 percentage points more likely to enroll in college.” 


2. The expected improvement in outcomes from attending a school with high promotion power is similar to the results that Deutsch et al. 
(2020) found for Louisiana high schools. They found that moving from an average school to a school at the 95th percentile increased 
the likelihood of high school graduation by 14 percentage points and college enrollment by 13 percentage points. The findings of the 
current study are also within the range of estimates found in research that used other approaches to estimate high school effectiveness. 
For example, a study using randomized admissions lotteries found that attending Promise Academy charter schools in New York City 
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Figure 2. Students attending public high schools in the District of Columbia (DC) with high promotion power are 
expected to have better outcomes than similar students attending schools with average or low promotion power 
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Note: n = 3,568 first-time grade 9 students in 36 public DC high schools during the 2015/16 school year. The blue bar (attending average school) rep- 
resents the student-level mean of the outcome among all students in the analytic sample. The orange bar (attending 90th percentile school) is the sum 
of the student-level mean outcomes and the school estimates at the 90th percentile of the promotion power distribution. The black bar (attending 10th 
percentile school) is the sum of the student-level mean outcomes and the school estimates at the 10th percentile of the promotion power distribution. 
Each set of bars in this figure represents different schools. It is unlikely that the same schools would be at the average, 90th, or 10th percentile of pro- 
motion power for each outcome. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


The expected outcomes for average students attending schools at the 10th percentile were worse than those for 
students attending schools with average promotion power. An average student who attended a high school at 
the 10th percentile for each outcome had an 8-15 percentage point lower chance of achieving that outcome. This 
pattern of results is not tautological or ensured by the model design; that is, the promotion power scores could 
have instead been clustered close together and not statistically distinguishable from each average. 


The promotion power models might be less successful in removing bias for some schools, and several of those 
schools are toward the higher and lower ends of the score distributions. For example, two of the three schools 
with the highest promotion power scores for college-ready SAT scores had selective admissions processes. Stu- 
dents in these schools might be particularly motivated or advantaged in unobserved ways. Furthermore, one 
school with an especially low promotion power score for college enrollment was an alternative school, which 
served students age 16 and older who fell behind grade level in high school credits. Even so, an analysis of schools 
at other points in the distribution, such as the 25th and 75th percentile, led to similar conclusions (see figure 1 for 
the full distribution of promotion power scores for each outcome). 


At least half of high schools had promotion power scores that were statistically different from the average for each 
outcome. For 69 percent of high schools, promotion power scores for college-ready SAT scores were statistically 
different from the average (table 3). Some 17 percent of high schools had promotion power scores for SAT scores 
that were statistically higher than the average (both the upper and lower black bars in figure 1 are above O), and 
53 percent had scores that were statistically lower than the average (both black bars in figure 1 are below 0). 
The proportion of schools that were statistically different from the average is somewhat lower for the other two 


increased the likelihood of high school graduation by 22 percentage points and college enrollment by 28 percentage points (Dobbie & 
Fryer, 2015). In Chicago, attending a Noble Street Charter School increased the likelihood of college enrollment by 13 percentage points 
(Davis & Heller, 2019). Using a similar research design, a study of outcomes for students attending small high schools in New York City 
found an impact of 9 percentage points on high school graduation and 8 percentage points on college enrollment (Unterman, 2014). 
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Table 3. Percentage of public high schools in the District of Columbia (DC) whose promotion power scores 
were statistically different from the average, by outcome 


College-ready High school College 
Measure SAT scores graduation enrollment 
Percentage of schools different from average 69 50 50 
Percentage of schools higher than average 7 31 25 
Percentage of schools lower than average 53 19 25 


Note: n = 3,568 first-time grade 9 students in 36 DC public high schools during the 2015/16 school year. All differences are statistically significant at 
p<.05. The sum of the percentage of schools higher than average (row 2) and the percentage of schools lower than average (row 3) might not equal the 
percentage of schools different from average (row 1) because of rounding. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


outcomes: 50 percent of schools had promotion power scores that were statistically different from the average 
for both high school graduation and college enrollment, with 31 percent statistically higher than average for high 
school graduation and with schools evenly distributed above and below the average for college enrollment. 


Schools with high promotion power for high school graduation were also more likely to have high 
promotion power for college enrollment 


This section examines the relationship between high schools’ power to promote different student outcomes. The 
correlation between a school’s promotion power for high school graduation and its promotion power for college 
enrollment is .66 (figure 3, panel a). Although the correlation between promotion power scores for these two 
outcomes is strong, it is not close to 1, suggesting that some schools are better at promoting one outcome than 
the other. The correlation also indicates the importance of using both measures in determining a high schools’ 
overall effectiveness, because they identify different ways that schools contribute to students’ long-term success. 


The correlation between a school’s promotion power for college-ready SAT scores and its promotion power for 
college enrollment is .59, nearly twice as high as the (statistically insignificant) correlation between a schools’ 
promotion power for college-ready SAT scores and for high school graduation (.32; see figure 3, panels b and c).3 
The difference between these correlations could be driven by the fact that the SAT score threshold is designed to 
signal that students are ready for college, which might not be the case for all high school graduates. Schools that 
improve students’ college readiness, as indicated by their SAT score, might also be better at improving students’ 
college enrollment. 


Student background characteristics were less strongly related to promotion power scores than to 
status measures such as high school graduation rate and college enrollment rate 


A primary goal of calculating promotion power scores is to separate a school’s contributions to student out- 
comes from the contributions of the background characteristics of the students it serves. This section assess- 
es how promotion power scores compare to status measures such as high school graduation rate and college 
enrollment rate in their relationship to student background characteristics, in this case the achievement level 
of a high school’s incoming students. The study used grade 8 standardized test scores as a summary measure of 


3. A small relationship between promotion power scores for college-ready SAT scores and high school graduation is consistent with 
recent studies comparing school and teacher effectiveness along test and nontest score outcomes. For example, one study found 
correlations of .24-.48 between test score value-added for Chicago high schools and value-added along other dimensions, such as 
grit, emotional health, academic engagement, school connectedness, and study habits (Jackson et al., 2020). Examining teacher value- 
added in the Los Angeles Unified School District, another study found an even weaker relationship between effectiveness at raising 
test scores and other measures, such as grade point average, absences, discipline, and grade retention (Petek & Pope, 2018). 
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Figure 3. The correlation between promotion power scores suggests that some public District of Columbia (DC) 
schools are better at promoting one outcome than another 
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c. College-ready SAT scores and high school graduation 
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Promotion power score for college-ready SAT scores 
*Statistically significant at p< .05. 


Note: n = 3,568 first-time grade 9 students in 36 DC public high schools during the 2015/16 school year. Each blue dot represents an individual high 
school. The orange line represents the line of best fit. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


background characteristics; students in high schools with lower average grade 8 standardized test scores could 
also be more disadvantaged in other ways. 


There is a nearly one-to-one relationship (a correlation of .92) between the grade 8 standardized test scores of a 
school’s students and the proportion of students who met the college-ready SAT benchmarks (figure 4, panel a). In 
contrast, there is a weaker relationship (.79) between grade 8 test scores and a school’s promotion power score for 
college-ready SAT scores (figure 4, panel b). The difference between these two correlations is statistically significant. 


The results for high school graduation and college enrollment reveal a similar pattern. Schools that served stu- 
dents with higher grade 8 standardized test scores were more likely to have higher overall graduation rates and 
college enrollment rates (see figure 4, panels c and e). However, the relationship between grade 8 test scores and 
promotion power for these outcomes, particularly for high school graduation, was weaker (see figure 4, panels d 
and f). For both outcomes the differences between the correlations are statistically significant. Therefore, for 
each outcome, grade 8 student achievement is less strongly related to promotion power scores than to tradition- 
al status measures. 
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Figure 4. Correlation between average grade 8 test scores of a school’s students and high school status 
measures and promotion power scores in public high schools in the District of Columbia (DC), by outcome 
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c. Grade 8 test scores and school-average high school 
graduation rate (status measure) 
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e. Grade 8 test scores and school-average college 
enrollment rate (status measure) 
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b. Grade 8 test scores and promotion power score 
for college-ready SAT scores 
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d. Grade 8 test scores and promotion power score for 
high school graduation 
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Grade 8 standardized test scores 


f. Grade 8 test scores and promotion power score for 
college enrollment 
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Grade 8 standardized test scores 


Note: n = 3,568 first-time grade 9 students in 36 DC public high schools during the 2015/16 school year. Each blue dot represents an individual high school. 
The orange line represents the line of best fit. Grade 8 standardized test scores represent the average of grade 8 standardized math and English language 
arts scores. For students who were missing either math or English language arts scores, the standardized score from the nonmissing subject was used. 


Source: Authors’ analysis of data from the DC Office of the State Superintendent of Education. 


REL 2021-098 


Limitations 


Promotion power models have two important limitations. First, no direct evidence has confirmed that promo- 
tion power scores are unbiased. Although several studies have found that school value-added measures contain 
limited, if any, bias (Angrist et al., 2016; Deming, 2014; Deutsch, 2012), there is reason to suspect that the concerns 
with traditional value-added measures become more pronounced with promotion power. For example, student 
background characteristics (including baseline test scores) in promotion power models explain less of the varia- 
tion in students’ long-term outcomes than is typically found for test scores the following year in traditional val- 
ue-added models (see table A4 in appendix A). And unobserved factors such as students’ motivation and family 
engagement, which likely correlate with the schools that students attend, could contribute more heavily to long- 
term outcomes. In fact, as shown in appendix B, the correlation between promotion power scores and school- 
average outcomes is considerably higher than the correlation between teacher value-added and average student 
test scores (Chetty et al., 2014). Even so, promotion power scores are still likely to represent an improvement over 
status measures, which do not account for student background characteristics at all. A further study that uses 
data from school admission lotteries (see, for example, Angrist et al., 2016; Deming, 2014; Deutsch, 2012) could 
empirically test for bias in promotion power scores. 


A second limitation of promotion power is that it does not reflect a school’s current effectiveness. Schools begin 
to influence student outcomes, such as high school graduation and college enrollment, as early as grade 9. For 
example, a grade 9 student facing hardship at home might receive extra support from a counselor, which increas- 
es the chances of that student staying on track in classes and ultimately graduating. Yet promotion power models 
do not capture this influence until several years later. Even when the focus is on SAT scores in grade 11, there 
is at least a three-year lag between when students first enter high school and when they take the SAT. The lag 
between when students begin to be influenced by their high schools and when promotion power models can 
capture that influence might be especially important if school effectiveness changes substantially in short periods 
of time. However, recent evidence finds that promotion power is stable over time. The correlation between the 
effectiveness of Louisiana high schools in one year and the next ranges from .63 to .73, depending on the outcome 
(Deutsch et al., 2020). Furthermore, traditional status measures such as high school graduation rates and college 
enrollment rates also suffer from a lag in outcome measurement. 


In addition to limitations of promotion power more broadly, there are also several limitations particular to this 
study. First, the study accounted for a somewhat limited set of student background characteristics compared 
with typical value-added and promotion power models; that could, in theory, create bias. Specifically, because 
of data limitations, the study did not include controls for student attendance or for student eligibility for the 
National School Lunch Program (an indicator of economic disadvantage) in middle school. Therefore, there could 
be differences between the school effects estimated in this study and what would have been estimated from a 
model with richer control variables. The study addressed this concern by conducting supplemental analyses that 
included administrative data from the Louisiana Department of Education (LDOE; see appendix B). LDOE records 
enabled the study team to compare promotion power models that included the limited set of controls from the 
main analysis with models that accounted for a broader set of controls. The supplemental analyses revealed that 
these two sets of controls correlate highly with each other, at least in the Louisiana context (see table BS in 
appendix B). 


Second, some school estimates might diverge further from the school’s true promotion power because the data 
included only a single cohort of grade 9 students. Even though the model used for the current study applied empir- 
ical Bayes shrinkage to enhance stability (see appendix A), one study shows that precision increases with multiple 
cohorts (Deutsch et al., 2020). The study team again used LDOE data, this time to compare models using one grade 
9 cohort and those using two grade 9 cohorts. Although the estimates from models with multiple cohorts were 
more precise, they correlate strongly with those from models with one cohort (see table B6 in appendix B). 
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Finally, the study did not observe outcomes for students who exited public DC high schools. That is, it did not 
distinguish between students who dropped out of high school and those who exited public DC high schools and 
enrolled in a high school outside the District of Columbia or a private high school in the District of Columbia. 
Thus, the analysis treats students who exited public DC high schools as if they did not meet the college and career 
readiness benchmarks on the SAT, did not graduate high school, and did not enroll in college. The measures might 
therefore penalize certain schools if higher-achieving students exited public DC schools. However, descriptive 
analyses in appendix B indicate that the students who exited public DC high schools (as proxied by whether stu- 
dents had available SAT scores) had lower grade 8 test scores and grade 9 attendance rates and higher grade 9 
discipline rates (see table B2 in appendix B). Therefore, it appears unlikely that many students who exited would 
have met the SAT benchmarks, graduated high school, and enrolled in college, so any bias from treating these 
students as not meeting these outcomes is likely small. 


Implications 


Promotion power scores meaningfully distinguished differences in high schools’ effectiveness at 
promoting student outcomes 


The study findings indicate that promotion power scores meaningfully distinguished among public DC high 
schools. After accounting for student background characteristics, the study found that some schools appeared 
especially effective at promoting college-ready SAT scores, high school graduation, and college enrollment, while 
others appeared to fall short. Therefore, for OSSE and other state education agencies and school districts, promo- 
tion power scores show promise for distinguishing differences in high school effectiveness. These measures can 
help identify low-performing schools that need support and high-performing schools that might serve as models 
for improving student outcomes. 


Furthermore, the study found that student background characteristics are less strongly related to promotion 
power scores than to status measures such as high school graduation rate and college enrollment rate. For this 
reason, accountability systems that use promotion power scores give schools that serve disadvantaged students 
a greater opportunity to demonstrate their effectiveness than systems that rely on status measures. 


Although schools’ promotion power scores were correlated across student outcomes, schools 
appeared to be differentially effective at promoting different outcomes 


Although the promotion power scores generally were correlated across outcomes, they were not so highly 
correlated as to suggest that they were duplicative. High schools in DC have differential power to promote col- 
lege-ready SAT scores, high school graduation, and college enrollment (that is, some schools are better at pro- 
moting one outcome than other outcomes). So, districts using promotion power scores for school accountability 
should include multiple outcomes to identify several important dimensions of school effectiveness. 


Differences in high schools’ power to promote different outcomes could also indicate that some outcomes are 
more objective than others and thus are better suited for use in an accountability system. Public DC high schools 
have some discretion to set more or less rigorous graduation requirements than the common set of requirements 
established by OSSE. Charter schools can impose additional requirements. In the other direction both district-run 
and charter schools can lower graduation requirements through waivers. This might explain the modest correla- 
tion between promotion power scores for high school graduation and college-ready SAT scores. 
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The District of Columbia Office of the State Superintendent of Education has a valuable opportunity 
to assess and enhance the validity of promotion power models by using data from the citywide public 
school admissions lottery 


Employing information from a school admissions lottery would allow the creation of promotion power scores that 
are demonstrably fair even without rich information about student background characteristics and preparation. 
My School DC (DC’s school lottery system) conducts an annual randomized admissions lottery to allocate seats in 
public schools around the city, both DC-operated schools and charter schools. Data from the randomized admis- 
sions lottery could be used, in a modified version of the promotion power models developed here, to create rig- 
orous measures of schools’ promotion power. The lottery allows the identification of students who are alike in all 
respects except that some won the lottery (for a slot in a particular school) and others did not (see Angrist et al., 
2017, for details on how lottery data can be used to improve school measures). 


More research is required to understand why some public District of Columbia high schools have 
much higher promotion power than others 


Although this study found that promotion power scores are useful for identifying more effective or less effec- 
tive high schools, it did not investigate the drivers of differential effectiveness across schools. Future research 
could identify the factors that make some schools more effective at promoting these student outcomes—such as 
school policies, programs, curricula, peer effects, or level of student motivation—both in public DC high schools 
and more broadly. Such research could be important to identify the levers to improve student outcomes. 
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